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Description 

Method and Arrangement for 

Microscopy 

Cross Reference to Related Applications 

[0001] This application claims priority of the German patent ap- 
plication 102 35 656.4 which is incorporated by reference 

herein. 
Background of Invention 

[0002] j ne invention concerns a method for microscopy. 

[0003] jhe invention further concerns an arrangement for mi- 
croscopy, the arrangement encompassing at least one mi- 
croscope objective, a detector unit for acquiring images of 
a sample, a display for displaying the images of the sam- 
ple acquired by the detector unit, and a computer system 
for controlling the microscope and the data acquisition. 

[0004] | n microscopy and in confocal microscopy, the user him- 
or herself has hitherto been forced to define the parame- 
ters that are necessary for three-dimensional recording of 



a structure. This is often very difficult or almost impossi- 
ble for an inexperienced user, since he or she does not 
have the necessary conceptual abilities. Even an experi- 
enced user often cannot locate structures within a sample 
on the basis of optical sections, resulting in incomplete 

recording of the desired structure. 
Summary of Invention 

[0005] it is the object of the invention to create a method which, 
by extensive automation, increases the user-friendliness 
and ergonomics of a microscope system and optimizes 
the three-dimensional recording of specimens. 

[0006] This object is achieved by way of a method comprising the 
steps of: 

[0007] a)acquiring an image and identifying a region of interest 
of a sample within the image; and 

[0008] b)automatically recording the entire marked sample re- 
gion in three dimensions. 

[0009] it is the object of the invention to create a method which, 
by extensive automation, increases the user-friendliness 
and ergonomics of a microscope system and optimizes 
the three-dimensional recording of specimens. 

[001 0] This object is achieved by way of an arrangement com- 
prising: a microscope with at least one microscope objec- 



tive, a detector unit for acquiring images of a sample, a 
display for displaying the images of the sample acquired 
by the detector unit, a computer system controlling the 
microscope and a data acquisition process, a means for 
identifying the region of interest of the sample, and in the 
computer system a means for automatically recording the 
entire marked sample region in three dimensions. 
1 ] It is particularly advantageous that a means for determin- 
ing the region of interest of a sample is provided. Further- 
more, the computer system encompasses a means for au- 
tomatically recording the entire marked sample region in 
three dimensions. The entire three-dimensional extension 
of a selected structure can thus be accomplished auto- 
matically, and the user does not necessary need, for that 
purpose, knowledge of the spatial extension of the struc- 
ture, and requires only minimal three-dimensional con- 
ceptual ability. The user can mark the structure of inter- 
est, for example, with a crosshairs, the crosshairs being 
overlaid on the image of the sample on the display. An- 
other advantageous solution is that the means for deter- 
mining the region of interest is a mouse cursor which can 
be moved around the region of interest of the sample on 
the display. 



[0012] The method is advantageous because it makes possible 

automatic recording of the entire marked sample region in 
three dimensions. Extraction of the region of interest of a 
sample from a sectional image is accomplished by image 
analysis, and from that a list of specimen positions within 
the sectional image is constructed. A list of possible fur- 
ther specimen positions outside the sectional image is 
then defined. That list is successively tested by traveling 
to the possible specimen positions with the microscope, 
imaging them, and testing whether they belong to the 
specimen. For that purpose, a sub-list of possible further 
specimen positions that can be imaged in parallel fashion 
by the microscope is sequentially selected. Based on that 
list, a microscope position is traveled to that makes pos- 
sible acquisition of a scene in which all the possible speci- 
men positions of the sub-list are visible. After a data ac- 
quisition has been performed, the sub-list of possible 
specimen positions is matched to the acquired data, all 
possible specimen positions not belonging to the speci- 
men are consequently eliminated, and the specimen posi- 
tions and intensity values belonging to the specimen are 
stored. 

[0013] Further advantageous embodiments of the invention are 



evident from the dependent claims. 
Brief Description of Drawings 

[0014] The subject matter of the invention is depicted schemati- 
cally in the drawings and will be described below with ref- 
erence to the Figures, in which: 

[0015] FIG. 1 is a schematic depiction of a scanning microscope; 

[0016] FIG. 2 shows a first variant of the selection of a desired 

structure on a display; 
[0017] FIG. 3 shows a second variant of the selection of a desired 

structure on a display; 
[0018] FIG. 4 schematically depicts the desired structure, in three 

dimensions, in a sample volume; 
[0019] FIG. 5a graphically depicts a first plane that intersects the 

desired structure; 
[0020] FIG. 5b graphically depicts a second plane that intersects 

the desired structure; 
[0021] FIG. 5c graphically depicts a third plane that intersects the 

desired structure; and 

[0022] FIG. 5d graphically depicts a fourth plane that intersects 

the desired structure. 
Detailed Description 

[0023] FIG. 1 schematically shows an exemplary embodiment of a 



confocal scanning microscope 100. This is not to be con- 
strued as a limitation of the invention. It is sufficiently 
clear to one skilled in the art that the invention can also 
be implemented using a conventional microscope system. 
Illuminating light beam 3 coming from at least one illumi- 
nation system 1 is directed by a beam splitter or a suit- 
able deflection means 5 to a scanning module 7. Before il- 
luminating light beam 3 strikes deflection means 5, it 
passes through an illumination pinhole 6. Scanning mod- 
ule 7 comprises a gimbal-mounted scanning mirror 9 that 
guides illuminating light beam 3 through a scanning opti- 
cal system 12 and a microscope objective 13 over or 
through a sample 15. In the case of non-transparent 
specimens 15, illuminating light beam 3 is guided over 
the specimen surface. With biological samples 15 
(preparations) or transparent specimens, illuminating light 
beam 3 can also be guided through sample 15. For that 
purpose, non-luminous preparations are optionally pre- 
pared with a suitable dye (not depicted, since established 
existing art). The dyes present in the specimen are excited 
by illuminating light beam 3 and emit light in a character- 
istic spectral region peculiar to them. This light proceed- 
ing from sample 15 defines a detected light beam 17. The 



latter travels through microscope optical system 13 and 
scanning optical system 12 and via scanning module 7 to 
deflection means 5, passes through the latter and arrives, 
through a detection pinhole 18, at at least one detector 
unit 19, which is embodied as a photomultiplier. It is clear 
to one skilled in the art that other detection components, 
for example diodes, diode arrays, photomultiplier arrays, 
CCD chips, or CMOS image sensors, can also be used. De- 
tected light beam 17 proceeding from or defined by sam- 
ple 15 is depicted in FIG. 1 as a dashed line. In detector 
19, electrical detected signals proportional to the power 
level of the light proceeding from specimen 15 are gener- 
ated. Since, as already mentioned above, light of more 
than one wavelength is emitted from sample 15, it is use- 
ful to insert in front of the at least one detector unit 19 a 
selection means 21 for the spectrum proceeding from 
specimen 15. The data generated by detector unit 19 are 
forwarded to a computer system 23. At least one periph- 
eral unit 27 is associated with computer system 23. One 
peripheral unit is embodied as a display 32 on which the 
user receives instructions for adjusting the scanning mi- 
croscope and can view the current setup as well as the 
image data in graphical form. Also associated with com- 



puter system 23 is an input means comprising, for exam- 
ple, a keyboard 28, an adjusting apparatus 29 for the 
components of the microscope system, and a mouse 30. 
[0024] FIG. 2 depicts display 32 with a structured image 33 of 

specimen 15 reproduced on display 32. Structured image 
33 of sample 15 represents a plane in sample 15 onto 
which microscope objective 13 is presently focused. In 
continuous scanning mode the user selects, for example, 
this one plane which shows him or her desired structure 
34. The user is optionally assisted in the search for de- 
sired structure 34 by autofocus and autogain settings that 
the microscope system comprises per se. A crosshairs 35 
is overlaid in display 32 for targeting onto desired struc- 
ture 34. The user can thereby obtain, from structured im- 
age 33 of sample 15, the first image that contains desired 
structure 34. The computer system encompasses a means 
25 for automatically recording the entire marked speci- 
men region. Means 25 for automatically recording the en- 
tire marked specimen region in three dimensions can be 
configured in the form of hardware and software. In a fur- 
ther embodiment, means 25 for automatically recording 
the entire marked specimen region in three dimensions is 
implemented in the form of a software module. 



[0025] FIG. 3 depicts another exemplary embodiment for select- 
ing a desired structure 34. In continuous scanning mode 
the user selects, for example, this one plane which shows 
him or her desired structure 34. In order to mark the de- 
sired structure 34, the user can draw a line 36 around de- 
sired structure 34. Continuous line 36 can be drawn us- 
ing, for example, the mouse or a joystick provided for the 
purpose. If the mouse is used for drawing, a drawing 
function can be assigned to a mouse cursor 37 by way of 
a click button on display 32. 

[0026] As already discussed in the description with reference to 
FIGS. 2 and 3, desired structure 34 or the region of inter- 
est of the sample is selected by the user. The microscope 
system then performs an automatic recording of the entire 
marked specimen region in three dimensions, i.e. the se- 
lected structure, and only that one, is recorded in all three 
spatial directions X, Y, and Z. In FIG. 4, desired structure 
34 selected by the user is depicted in an enclosing sample 
volume 40. Sample volume 40, and thus selected struc- 
ture 34 as well, is intersected by multiple, equally spaced 
planes 42 , 42 , 42 , and 42 . Planes 42 , 42 , 42 , and 
42 4 are parallel to XY plane 41 of sample volume 40. The 
user selects a plane within sample volume 40. The se- 



lected plane corresponds to the plane onto which the mi- 
croscope system is focused, and which contains the re- 
gion of interest of the sample depicted on display 32. Ex- 
traction of the region of interest, or of the selected struc- 
ture, of a sample is accomplished by image analysis. A list 
containing the specimen positions is constructed. The 
specimen positions are determined by way of the XY coor- 
dinates in the image frame. The location of the planes and 
their spacing results from the optimum Z resolution of 
microscope objective 13 in use at the time. By way of an 
appropriate code on the usable objectives, computer sys- 
tem 23 of the microscope system can automatically per- 
form the calculation of the optimum Z resolution of mi- 
croscope objective 13. Acquisition of a scene is achieved 
using means 7, 38, 39 for arriving at the microscope po- 
sition. The means for arriving at a possible microscope 
position is made possible by a displacement device 38 for 
the microscope objective in the Z direction and a dis- 
placement device 39 for the microscope stage in the XY 
plane. In a scanning microscope, the means for arriving at 
a possible microscope position is implemented by way of 
a displacement device 38 for microscope objective 13 in 
the Z direction and scanning module 7, which scans sam- 



pie 15 in the XY plane. A combination of scanning module 
7, displacement device 38 for microscope objective 13, 
and displacement device 39 for the microscope stage is 
also conceivable for arriving at a desired microscope posi- 
tion. 

[0027] FIGS. 5a through 5d depict, in plan view, the individual 
planes 42 , 42 , 42 , and 42 of FIG. 4. Planes 42 , 42 , 
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42 3 , and 42 4 are subdivided into multiple pixels 45 of 
identical size. Pixels 45 have a finite extension and repre- 
sent the size of the successive scan regions. In FIGS. 5a 
through 5d, pixels 45 are depicted as two-dimensional. It 
is self-evident, however, that the pixels can also be three- 
dimensional voxels. The user has, for example, focused 
on third plane 42 3 , so that initially one possible specimen 
position 50 has been found in this plane by means of 
suitable image processing. Proceeding from the possible 
specimen position 50, further specimen positions 51 are 
searched for in the immediate vicinity around the possible 
specimen position. The search is initially confined only to 
the selected plane 42^. From the search, a list of possible 
further specimen positions is generated. These further 
specimen positions 51 are then labeled with reference 
character 51 only if, for further specimen position 51, at 



least a portion of the selected structure is located within a 
pixel. The procedure continues accordingly on the basis of 
further specimen positions 51, and additional specimen 
positions 52, which likewise contain at least one pixel of 
the desired structure, are generated. The conditions for 
labeling with reference character 52 are the same as for 
the specimen positions labeled 51. Proceeding from spec- 
imen positions 52, specimen positions 53 are searched 
for. This continues until no further pixels containing at 
least a portion of the selected structure can be found in 
the selected plane 42^ All of the specimen positions that 
contain a portion of selected structure 34 are entered in a 
sub-list of specimen positions that can be acquired in 
parallel fashion by the microscope. In compiling the sub- 
list, care is taken that none of the pixels having a desired 
structure 34 is counted twice. The last step is to travel to 
a microscope position which allows acquisition of a scene 
in which all possible specimen positions on the sub-list 
are visible. Data acquisition is then performed in the se- 
lected plane 42 . 
K 3 

[0028] As already schematically depicted in FIG. 4, desired struc- 
ture 34 extends not only in plane 42 3 that has already 
been examined, but also onto planes 42 and 42 . In 

1 2 



plane 42^, the method finds no pixels in which at least 
some of the desired specimen positions 50, 51, 52, and 
53 are located. Based on specimen positions 50, 51, 52, 
and 52 found in plane 42^ the list of possible specimen 
positions is expanded into planes 42^ and 42^ Proceed- 
ing from the specimen position, the method or micro- 
scope system searches in plane 42 for further possible 

4 

specimen positions. Since plane 42 (FIG. 5d) contains no 

4 

further possible specimen positions, all that is labeled 
here is possible specimen position 60, located opposite 
specimen position 50 from plane 42^ The search for pos- 
sible specimen positions in this plane can be discontin- 
ued, since no further specimen positions containing de- 
sired structure 34 are found. 
[0029] when the search is also extended to plane 42 2 lying above 
plane 42 3 , a pixel having specimen position 70, which 
corresponds to specimen position 50 of plane 42 3 , is se- 
lected. Since a pixel containing at least a portion of de- 
sired structure 34 is found here, the number 70 is entered 
for that pixel. The search for possible further specimen 
positions is performed as already described above, by ex- 
amining the nearest neighbors starting at possible speci- 
men position 70. As a result, further specimen positions 



71, 72, 73, and 74 that contain a portion of desired struc- 
ture 34 are found in plane 42 2 _ In searching for the near- 
est neighbors that contain at least a portion of the desired 
structure, it is important to ensure that double counting 
does not occur, i.e. that no specimen positions already 
previously identified in the plane as a specimen position 
are entered in the list. 

[0030] Proceeding from plane 42 2 , the pixel having specimen po- 
sition 70 is selected, and the search is extended from 
there in plane 42^ That pixel is selected as the starting 
point of a possible specimen position 80. Proceeding from 
specimen position 80, further possible specimen positions 
81, 82, and 83 are identified. 

[0031] it j S self-evident that in the search for possible specimen 
positions, when the method or the microscope system 
finds a specimen position in a plane it also examines the 
specimen positions immediately adjacent to that plane. 

[0032] The test implicit in these depictions, as to whether or not 
a pixel belongs to a specimen structure, can represent an 
entire class of functions from the field of image process- 
ing. In the simplest expression, a class is formed byway 
of the acquired intensity of the measurement point. As 
soon as the acquired value is greater than a threshold 



value T, the pixel is considered to belong to the specimen; 
otherwise it is discarded. This simplest decision is already 
sufficient for applications in fluorescence microscopy. In 
an advantageous expression of the invention, that thresh- 
old value is defined by the user. In an even more advanta- 
geous expression, the starting image is tested for multi- 
modality, and an upper and a lower threshold value are 
defined based on the values. Computer-assisted methods 
that do this are, for example, the Otso method or recur- 
sively operating entropy maximization for grayscale im- 
ages. The literature concerning pattern recognition (e.g. 
Duda, Hart, Pattern Classification, Wiley) discloses further 
mechanisms that function similarly; these are cited as ex- 
isting art and regarded as interchangeable. 

[0033] An even more advantageous expression adapts the selec- 
tion of the threshold value by way of the data acquisition, 
the methods described above being substantially per- 
formed over adjacent previously-acquired test positions 
and weighted by comparison to the initial value. 

[0034] For non-fluorescent microscopy, it may be advantageous 
to perform a data preprocessing operation by way of the 
intensities of the test positions in order to achieve an 
ideal separation using the method described above. The 



type of preprocessing depends greatly on the image 
modality and the contrast methods used. 



